Deep Neural Network-Based Decision Network

ABSTRACT

The technology disclosed proposes using a combination of computationally cheap, less-accurate bag of words (BoW) model and computationally expensive, more-accurate long short-term memory (LSTM) model to perform natural processing tasks such as sentiment analysis. The use of cheap, less-accurate BoW model is referred to herein as “skimming”. The use of expensive, more-accurate LSTM model is referred to herein as “reading”. The technology disclosed presents a probability-based guider (PBG). PBG combines the use of BoW model and the LSTM model. PBG uses a probability thresholding strategy to determine, based on the results of the BoW model, whether to invoke the LSTM model for reliably classifying a sentence as positive or negative. The technology disclosed also presents a deep neural network-based decision network (DDN) that is trained to learn the relationship between the BoW model and the LSTM model and to invoke only one of the two models.

CROSS REFERENCE TO OTHER APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/471,934, entitled “LEARNING WHEN TO SKIM AND WHEN TOREAD,” filed on Mar. 15, 2017, (Atty. Docket No. SALE 1190-1/3054PROV).The provisional application is hereby incorporated by reference for allpurposes; and

This application incorporates by reference U.S. Nonprovisional patentapplication Ser. No. ______ entitled “PROBABILITY-BASED GUIDER” (Atty.Docket No. SALE 1190-2/3054US1), filed contemporaneously herewith.

FIELD OF THE TECHNOLOGY DISCLOSED

The technology disclosed relates to artificial intelligence typecomputers and digital data processing systems and corresponding dataprocessing methods and products for emulation of intelligence (i.e.,knowledge based systems, reasoning systems, and knowledge acquisitionsystems); and including systems for reasoning with uncertainty (e.g.,fuzzy logic systems), adaptive systems, machine learning systems, andartificial neural networks. The technology disclosed uses deep neuralnetworks like long short-term memory (LSTM) networks for naturallanguage processing tasks.

BACKGROUND

The subject matter discussed in this section should not be assumed to beprior art merely as a result of its mention in this section. Similarly,a problem mentioned in this section or associated with the subjectmatter provided as background should not be assumed to have beenpreviously recognized in the prior art. The subject matter in thissection merely represents different approaches, which in and ofthemselves can also correspond to implementations of the claimedtechnology.

Recent deep learning advances in natural language processing (NLP) havebeen focused on one goal—state-of-the-art performance no matter thecomputational cost. We propose a deep learning-based solution thatreduces computational cost while retaining high accuracy provided bydeep neural networks. By letting a fast baseline model decide whichsentence is easy or difficult, we show that an inexpensive classifier,such as the bag of words, learns how to separate easy from difficultsentences. Our results find that thresholding probability (produced as abyproduct during classification) outperforms not having a guidedstrategy on accuracy versus time saved metric.

Deep learning models are getting bigger, better, and more expensive inthe all-or-nothing hunt for reaching superhuman performance. In naturallanguage processing (NLP), often new additions base their novelstate-of-the-art design on a relatively narrow extension that solves acomplex linguistic task really well. This includes incorporatingsemantic knowledge, reducing dictionary size, or even increasingdictionary size. Intuitively, it would make sense that such additionswould only exert the new system to those data points having the specificchallenges that the addition conquered. That is, however, often not thecase as deep learning algorithms are mostly benchmarked with aninvariant of being oblivious to the given data point and the difficultyhereof. Further, with the increased understanding that deeper meansbetter, current state-of-the-art systems can be excessive and wastecomputation to secure a narrow accuracy improvement in a linguistic areathat may not be relevant in most scenarios. For example, FIG. 3illustrates how a cheap model clusters sentences when it is trained forsentiment analysis (e.g., using backpropagation). In FIG. 3, someclusters are marked to show that some data points are easier than othersand that supervised learning can reveal clustering of certain linguisticproperties.

The technology disclosed proposes two different methods forunderstanding when a sentence is easy to classify and when it isdifficult. The technology disclosed applies either a cheap bag of words(skimming) or an expensive LSTM (reading) to classify a sentence basedon the decision of the methods. The first method uses the probabilityoutput of the bag of words system as a confidence measure. The secondmethod builds a decision neural network to learn the relationshipbetween when the bag of words might be correct and when the LSTM mightbe correct.

Both our methods outperform not having a guided baseline strategy, andcan save significant computation resources for engineering teams.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee. The color drawings also may be available in PAIRvia the Supplemental Content tab.

In the drawings, like reference characters generally refer to like partsthroughout the different views. Also, the drawings are not necessarilyto scale, with an emphasis instead generally being placed uponillustrating the principles of the technology disclosed. In thefollowing description, various implementations of the technologydisclosed are described with reference to the following drawings, inwhich:

FIG. 1 illustrates an architectural level schematic of a system in whicha probability-based guider automates selection of a machine learningmodel for performing a machine classification task. In oneimplementation, the machine classification task can be sentimentclassification of a sentence. The sentiment classification task cancomprise classifying a sentence as having a positive or negativesentiment. In other implementations, the sentiment classification taskcan comprise classifying the sentence as having a very positive,somewhat positive, neutral, very negative, or somewhat negativesentiment. In yet other implementations, additional, fewer, or differentclass labels can be used. In further implementations, other machineclassification tasks can be performed such as speech recognition,machine translation, text summarization, question answering, imagecaptioning, and text-to-speech (TTS) synthesis.

FIG. 2A is a flowchart illustrating process steps for classification ofinput sentences using the probability-based guider of FIG. 1.

FIG. 2B is a flowchart illustrating process steps for sentimentclassification task.

FIG. 3 is a graphical illustration of sentiment classification resultsusing bag of words (abbreviated BoW) model on input sentences from anexample database.

FIG. 4 is an example architecture of the BoW model.

FIG. 5 is an example architecture of long short-term memory (abbreviatedLSTM) model.

FIG. 6A is a graphical illustration of sentiment prediction results oninput sentences from an example database using the BoW model.

FIG. 6B illustrates probabilities assigned to predicted sentimentclasses in the output layer of the BoW model of FIG. 6A.

FIG. 7A is a graphical illustration of sentiment prediction results oninput sentences from an example database using the LSTM model.

FIG. 7B illustrates probabilities assigned to predicted sentimentclasses in the output layer of the LSTM model of FIG. 7A.

FIG. 8A is a graphical plot illustrating prediction accuracy of the BoWmodel of FIG. 6A in various probability intervals.

FIG. 8B is a graphical plot showing the amount of data per threshold binfor the BoW model.

FIG. 8C is a graphical plot similar to FIG. 8A showing predictionaccuracy of the LSTM model of FIG. 7A.

FIG. 8D is a graphical plot showing the amount of data per threshold binfor the LSTM model.

FIG. 9 illustrates an architectural level schematic of a system in whicha decision network is used for selection of a machine learning model forperforming a machine classification task. In one implementation, themachine classification task can be sentiment classification of asentence. The sentiment classification task can comprise classifying asentence as having a positive or negative sentiment. In otherimplementations, the sentiment classification task can compriseclassifying the sentence as having a very positive, somewhat positive,neutral, very negative, or somewhat negative sentiment. In yet otherimplementations, additional, fewer, or different class labels can beused. In further implementations, other machine classification tasks canbe used for training such as speech recognition, machine translation,text summarization, question answering, image captioning, andtext-to-speech (TTS) synthesis.

FIG. 10 illustrates the operation of a splitter of the system of FIG. 9to create a training database and a validation database.

FIG. 11 shows training of the candidate neural network models using thetraining database generated by the splitter as shown in FIG. 10.

FIG. 12 shows testing of the trained neural network models usingvalidation data generated by the splitter as shown in FIG. 10.

FIG. 13 illustrates the operation of the label transformer of the systemin FIG. 9 using the confusion matrix to generate validation data withnew model class labels replacing the task class labels.

FIG. 14 illustrates training of the decision network of FIG. 9 using thevalidation data with new model class labels.

FIG. 15 shows an example architecture of a decision network of FIG. 9.

FIG. 16 is a flowchart illustrating process steps for model selectionusing decision network of FIG. 9.

FIG. 17 is a simplified block diagram of a computer system that can beused to implement the machine learning system of FIG. 1 and FIG. 9.

DETAILED DESCRIPTION

The following discussion is presented to enable any person skilled inthe art to make and use the technology disclosed, and is provided in thecontext of a particular application and its requirements. Variousmodifications to the disclosed implementations will be readily apparentto those skilled in the art, and the general principles defined hereinmay be applied to other implementations and applications withoutdeparting from the spirit and scope of the technology disclosed. Thus,the technology disclosed is not intended to be limited to theimplementations shown, but is to be accorded the widest scope consistentwith the principles and features disclosed herein.

INTRODUCTION

The technical problem that the technology disclosed sets out to solve ishow to introduce computation efficiency in deep neural networks-basednatural language processing tasks, while maintaining the accuracyprovided by modern deep neural networks.

To this end, the technology disclosed proposes using a combination ofcomputationally cheap, less-accurate bag of words (BoW) model andcomputationally expensive, more-accurate long short-term memory (LSTM)model to perform natural processing tasks such as sentiment analysis.The use of cheap, less-accurate BoW model is referred to herein as“skimming”. The use of expensive, more-accurate LSTM model is referredto herein as “reading”.

The technology disclosed presents a probability-based guider (PBG). PBGcombines the use of BoW model and the LSTM model. PBG uses a probabilitythresholding strategy to determine, based on the results of the BoWmodel, whether to invoke the LSTM model for reliably classifying asentence as positive or negative. PBG uses the probability output of theBoW model as a confidence measure and subsequently invokes the LSTMmodel only when the probability output of the BoW model is below a setthreshold.

The technology disclosed also presents a deep neural network-baseddecision network (DDN) that is trained to learn the relationship betweenthe BoW model and the LSTM model (e.g., using backpropagation) and toinvoke only one of the two models. In other words, for a particularinput, the DDN selects the correct model from an ensemble of pre-trainedmodels, with the selection conditioned on minimizing the computationalcost and maximizing and/or preserving the classification accuracy. Inone implementation, DDN includes a two-layer multilayer perceptron (MLP)stacked on top of a BoW model, with recurring dropout layers, ReLUnonlinearities, and fully-connected layers.

Unlike PBG, DDN is a neural network-based model that can be trained(e.g., using backpropagation). The PBG always has to use the BoW modeland often uses both the BoW and the LSTM models. In contrast, the DDNmay decide to skip the BoW model and only use the LSTM model or juststop at the BoW model because it knows that the LSTM model is unlikelyto produce better results.

Any neural network-based model is only as good as the data available totrain it. Having a large training set prevents overfitting of the model.To this end, the technology disclosed uses the BoW and the LSTM modelsto automatically generate labeled training examples, with the labelsassigned based on a confusion matrix. The DDN is then trained on asecondary supervised classification task of choosing the correct modelfor classifying a given sentence. The confusion matrix identifies howoften the two models correctly and incorrectly classify a sentence aspositive or negative. The DDN is then trained on the automaticallylabeled sentences.

The technology disclosed then creates a new held-out split from theoriginal full training set. The full training set comprises a modeltraining set (e.g., 80% of training data) and a decision training set(remaining 20% of training data). First, the BoW and the LSTM models aretrained on the model training set to generate labels for the DDN (e.g.,using backpropagation). The DDN is then trained on the decision trainingset (e.g., using backpropagation). Lastly, the BoW and the LSTM modelsare fine-tuned on the full training set while holding the DDN fixed.This helps DDN generalize to models that are fine-tuned on the fulltraining set.

System Overview—Probability-Based Guider

We describe a system for sentiment classification of input sentences.The system and processes are described with reference to FIG. 1 showingan architectural level schematic of a system in accordance with animplementation. Because FIG. 1 is an architectural diagram, certaindetails are intentionally omitted to improve the clarity of thedescription.

The discussion of FIG. 1 is organized as follows. First, the elements ofthe figure are described, followed by their interconnections. Then, theuse of the elements in the system is described in greater detail.

FIG. 1 includes the system 100. The system 100 includes a machinelearning system 110, a network(s) 120, a training database 142, avalidation database 144, and a probability-based guider 132. In otherimplementations, the system 100 also includes a testing database withtesting data that is used to test the machine learning system 110 duringinference (production or testing).

The machine learning system 110 includes a bag of words (abbreviatedBoW) model 112 and a recurrent neural network (abbreviated RNN) model114. The training database 142 and the validation database 144 containsentiment analysis data such as natural language sentences with positiveand negative sentiment labels for each sentence. In one implementation,the training database 142 and the validation database 144 containEnglish language sentences and a positive or a negative sentiment labelfor each sentence. The BoW 112 and the RNN 114 separately perform thesentiment classification task on one sentence from the training database142 or the validation database 144 at a time. Both the BoW 112 and theRNN 114 can run a variety of hardware processors such as graphicprocessor unit (GPUs). Neural network-based models involvecomputationally intensive methods. GPUs are well suited for these typesof computations. Recently, specialized hardware is being developed toefficiently train neural network models.

A recurrent neural network (RNN), such as RNN 114, processesvariable-length sequence input. It uses a recurrent hidden state whoseactivation is dependent on that of the one immediate before. Moreformally, given an input sequence x=(x₁, x₂, . . . , x_(T)), a RNNupdates the hidden vector sequence h=(h₁, h₂, . . . , h_(T)) and outputvector sequence y=(y₁, y₂, . . . , y_(T)) from t=1 to T as follows:

h _(t) =H(W _(xh) x _(t) +W _(hh) h _(t-1) +b _(h))

y _(t) =W _(hy) h _(t) +b _(y))

In the mathematical formulation above, W denotes weight matrices, bdenotes bias vectors, and H is the recurrent hidden layer function.

Long short-term memory (LSTM), such as LSTM model 244, is a powerfulvariant on the RNN 114. A LSTM is a cell in a neural network that isrepeatedly exercised in timesteps to produce sequential outputs fromsequential inputs. The output is often referred to as a hidden state,which should not be confused with the cell's memory. Inputs are a hiddenstate and memory from a prior timestep and a current input. The cell hasan input activation function, memory, and gates. The input activationfunction maps the input into a range, such as −1 to 1 for a tan hactivation function. The gates determine weights applied to updating thememory and generating a hidden state output result from the memory. Thegates are a forget gate, an input gate, and an output gate. The forgetgate attenuates the memory. The input gate mixes activated inputs withthe attenuated memory. The output gate controls hidden state output fromthe memory. The hidden state output can directly label an input or itcan be processed by another component to emit a word or other label orgenerate a probability distribution over labels.

The input gate controls how much of the current input x_(t) and theprevious hidden state h_(t-1) enters the current memory cell state m_(t)and is represented as:

$\begin{matrix}{i_{t} = {\sigma \left( {{W_{xi}x_{t}} + {W_{hi}h_{t - 1}} + b_{i}} \right)}} \\{= {\sigma \left( {{{linear}_{xi}\left( x_{t} \right)} + {{linear}_{hi}\left( h_{t - 1} \right)}} \right)}}\end{matrix}$

The forget gate operates on the current memory cell state m_(t) and theprevious memory cell state m_(t) and decides whether to erase (set tozero) or keep individual components of the memory cell and isrepresented as:

f _(t)=σ(W _(xf) x _(t) +W _(hf) h _(t-1) +b _(f))

The output gate scales the output from the memory cell and isrepresented as:

o _(t)=σ(W _(xo) x _(t) +W _(ho) h _(t-1) +b _(o))

The LSTM can also include a current hidden state producer that outputsthe current hidden state h_(t) scaled by a tan h (squashed)transformation of the current memory cell state m_(t) and is representedas:

h _(t) =o _(t)□ tan h(m _(t))

In the above equation, □ represents element-wise product.

A bag of words (BoW) model, such as BoW 112, determines a mean of theinput words of a sentence across each feature dimension. In oneimplementation, the BoW model determines average or mean of wordembeddings (e.g., GloVe or word2vec). Taking a mean can be a cheapoperation and can reduce the dimensionality of the sentence to a fixedsized vector. Some examples of the BoW model or variants thereof includea continuous bag of words (CBoW) model and a skip-gram model proposed byTomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.Efficient Estimation of Word Representations in Vector Space. In ICLRWorkshop Papers; Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado,and Jeffrey Dean. 2013b. Distributed representations of words andphrases and their compositionality. In NIPS, pages 3111-3119; and TomasMikolov, Wen tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularitiesin continuous space word representations. In HLTNAACL, which areincorporated by reference for all purposes as if fully set forth herein.A CBoW model predicts a word from the sentence context. A skip-grammodel predicts the sentence context from the word.

During training of the BoW 112 and the RNN 114, the networks receiveinput sentences from the training database 142 and process it to performthe sentiment classification task. During validation, the BoW 112 andthe RNN 114 receive input sentences from the validation database 144 andprocess it to perform the sentiment classification task. The machinelearning system 110, the training database 142, the validation database144, and the probability-based guider 132 are in communication with eachother via the network(s) 120.

In other implementations, probability-based guider 132 can be used withother machine classification tasks such as speech recognition, machinetranslation, text summarization, question answering, image captioning,and text-to-speech (TTS) synthesis.

Having presented a high level description of the system 100, thediscussion now turns to detailed description of various components ofthe system 100.

Process Flow

FIG. 2A is a flowchart 200A illustrating the processing of inputsentences for sentiment classification task. The process starts at step210 when a sentence is given as input to the BoW model 112. At step 212,the BoW model 112 evaluates the input sentence and produces a confidencescore which specifies a likelihood of the sentence's sentiment beingpositive or negative. In one implementation, the value of the confidencescore is a real value between zero and one and can be preset (e.g.,0.5). In this implementation, a confidence score value of less than 0.5means that the input sentence belongs to a first sentiment class and aconfidence score of greater that 0.5 means that the sentence belongs toa second sentiment class. A confidence score value closer to the 0.5,i.e., 0.49 or 0.51 means that the BoW model 112 is not very confidentabout the sentiment of the input sentence as being positive or negative.Confidence score values that are farther away from 0.5, i.e., 1 or 0.9indicate that BoW model is very confident about sentiment of thesentence as being positive or negative.

At step 214, the value of the confidence score generated by the BoWmodel 112 is checked by the probability-based guider 132 against a setthreshold (e.g., 0.5). In one implementation, the threshold is a singlevalue. For example, for confidence score values between zero and one (asillustrated above), a threshold may be set at 0.4 for the firstsentiment class and 0.6 for the second sentiment class. In such animplementation, if the confidence score generated by the BoW model 112is below 0.4, the probability-based guider considers this as a confidentclassification of the sentence in the first sentiment class. Likewise, aconfidence score above 0.6 shows that the BoW model 112 is confident inclassifying the input sentence in the second sentiment class. It isunderstood that different values of the threshold can be selected basedon user-preferences and requirements.

The probability-based guider compares the confidence score produced bythe BoW model 112 against the set threshold. If the confidence score ishigher than the threshold based on comparison, then sentimentclassification of the sentence by the BoW mode is accepted at step 216.Otherwise, if the confidence score is lower than the threshold based oncomparison, then the sentence is given as input to the RNN model 114 atstep 218. The RNN model 114 then classifies the sentence's sentiment aspositive or negative. The sentiment classification produced by the RNNmodel 114 is accepted at step 220. At step 222, the system checks ifthere are more input sentences for sentiment classification. If this istrue, then the next sentence is given as input to the BoW mode at 212and the process is repeated. The process ends at step 224 when there areno further input sentences for sentiment classification.

FIG. 2B is a process flow 200B illustrating a sentence sentimentclassification task. The sentiment classification task can compriseclassifying a sentence as having a positive or negative sentiment. Inother implementations, the sentiment classification task can compriseclassifying the sentence as having a very positive, somewhat positive,neutral, very negative, or somewhat negative sentiment.

A sentence 230 is given as input to a neural network-based first naturallanguage processing (NLP) model such as the BoW model 112. The BoW model112 generates a bag of word sentiment classification probability “α”(read as alpha) at step 236. The probability-based guider 132 comparesthe value of “α” with the set threshold value “τ” (read as tau) todetermine whether the input sentence requires supplemental evaluation bya neural network-based second NLP model such as LSTM 244, which is moreaccurate and computationally more expensive than the first NLP model. Ifthe value of “α” is greater than the value of the threshold “τ” (step238), sentiment classification of the input sentence 230 by BoW model112 is accepted (step 240).

If the value of the classification probability “α” is less than thevalue of the threshold “τ” (step 242), the input sentence 230 is sent tothe long short-term memory (LSTM) model 244 for sentiment classificationtask. The LSTM 244 then generates a sentiment classification of thesentence which is accepted at step 246.

FIG. 3 illustrates innards 300 of the BoW model, classifying sentencesas positive or negative. One way of exploring deep learning models is byplotting how each sentence is represented in the hidden layers. However,as the hidden layers are often high dimensional, algorithms such ast-distributed stochastic neighbor embedding (T-SNE) are used to reducedimensionality to two-dimensions (2D) to allow human inspections. T-SNEis a machine learning algorithm for dimensionality reduction and allowsvisualizations of high dimensional data into a two or three dimensionalspace.

FIG. 3 is a visualization of the last hidden layer from a two-layeredmulti-layered perceptron (MLP) on top of a BoW model 112. The boundaryboxes in FIG. 3 offer some important insights. Real world data comes indifferent levels of difficulties; some sentences are easily classifiedwhile others contain complex semantic structures. Linguistically complexsentences have co-occurrences of negative and positive constituents.Additionally, linguistically complex sentences can have multipleinstances of negative words and contrastive conjunctions. Acomputationally less expensive model such the BoW model 112 canconfidently classify linguistically simple sentences. However, a lessexpensive model produces less confident classification score onlinguistically complex sentences. On the contrary, for linguisticallysimple sentences a computationally less expensive model such as the BoWmodel 112 can generate confident classification scores, and therefore acomputationally expensive model such as the LSTM 244 is not required.

Model Architecture

FIG. 4 is an architecture 400 of the BoW model 112. The word embeddingsin the bottom box are initialized with GloVe (Global Vectors for WordRepresentations). Other word embedding models such as Word2Vec can alsobe used to generate word embeddings. This is followed by calculation ofthe mean of the words (bag of words) and two layer MLP with dropout. Theoutput layer at the top is a classifier, generating a positive or anegative sentiment classification for the input sentence. This is onlyone implementation of the architecture 400. In other implementations,any other conventional or future-developed BoW architectures can be usedherein. Such implementations will be readily apparent to those skilledin the art without departing from the spirit and scope of the technologydisclosed.

FIG. 5 is an architecture 500 of the LSTM model 244. The word embeddingsin the bottom box are initialized with GloVe as above. Gradients are nottracked through the word embeddings. A bidirectional RNN is used withLSTM cells, in similar manner to the BoW model. Hidden states of RNN areused to extract mean and max values. This is followed by a two layer MLPwith dropout. Finally, the output layer classifies the sentiment of theinput sentence as positive or negative. This is only one implementationof the architecture 500. In other implementations, any otherconventional or future-developed RNN or LSTM architectures can be usedherein. Such implementations will be readily apparent to those skilledin the art without departing from the spirit and scope of the technologydisclosed.

Validation Results

The following figures illustrate the performance of the BoW model 112and the RNN model 114 on the validation database 144.

FIG. 6A is a T-SNE plot 600A of the last hidden layer in the BoW model112. Each data point in the plot 600A is a sentence. The data points arecolored. The center of a data point represents its prediction by the BoWmodel 112. Blue color means a sentence represents a positive sentimentand a red color indicates a sentence representing a negative sentiment.The color of the line around the center indicates the actual sentimentlabel of the sentence. This allows for inspection of when the BoW model112 has correctly predicted the sentiment of the sentence and when it iswrong.

A data point with a red colored center and a red colored borderindicates a true negative meaning the model correctly predicted anegative sentiment sentence. A data point with a red colored center andblue colored border is a false negative meaning the model incorrectlypredicted a positive sentiment sentence as a negative sentimentsentence. A data point with a blue colored center and a blue coloredborder indicates a true positive meaning the model correctly predicted apositive sentence. A data point with a blue colored center and a redcolored border indicates false positive meaning the model incorrectlypredicted a negative sentiment sentence as a positive sentimentsentence.

Analysis of the plot 600A shows that sentences fall into clustersaccording to their linguistic complexity. Some clusters lie along asimple manifold which are predicted with high confidence score by theBoW model 112. Other clustered are more scattered and are assigned lowconfidence scores by the BoW model 112. Further analysis of the sentencestructure indicated that sentences with both positive and negativeconstituent are difficult to predict accurately by the BoW model 112.

FIG. 6B shows a plot 600B of probability assigned to the predicted classin the output layer of the BoW model 112. The dark blue colored datapoints indicate high confidence of the model in sentence sentimentprediction while light colored data points indicate lower confidence ofthe model in sentence sentiment prediction.

FIG. 7A and FIG. 7B are validation results for LSTM model 244. FIG. 7Ais a plot 700A and is similar to plot 600A. We observe that plot 700Afor LSTM model 244 has relatively few data points with low confidence.The LSTM model 244 classifies the sentences containing positive andnegative sentiments with higher confidence. Plot 700B in FIG. 7Billustrates that LSTM model 244 correctly classifies most of thesentences (represented as data points). Dark blue colored dots indicatevery confident classification. Relatively fewer sentences are lightlycolored, indicating lower confidence of the model.

Probability Strategy

The BoW model 112 and the LSTM model 244 are trained (e.g., usingbackpropagation) to give a probability for each class (e.g., positivesentiment and negative sentiment) which indicates the confidence of themodel prediction. For a binary classification, in one implementation,the probability has to be over 0.5, otherwise, the prediction is for theother sentiment class. For example, if BoW model 112 returns 1, it meansit is very confident in its prediction. Thresholding means comparing thepredicted probability by the model and assessing whether to use it ornot. For example, a threshold of 0.7 means that sentiment classificationof all sentences with a probability of 0.7 or above can be accepted.Another approach to analyze the results of the models is to look at theintervals such as the sentences that were correctly predicted ininterval 0.5 to 0.55. This shows how accurate the predictions of a modelare in this confidence range.

FIG. 8A is a plot 800A illustrating prediction accuracy of the BoW model112 in various probability intervals between 0.5 and 1. The plot 800Ashows that increasing the probability threshold increases theperformance of the model. FIG. 8C is a similar plot 800C for the LSTMmodel 244. From the LSTM plot 800C it is not obvious that increasing theprobability threshold increases the performance of the model. This isbecause LSTM model 244 can overfit the training data and provide onlyconfident answers.

Using the probability threshold, the technology disclosed creates a“probability strategy”. The output probability gives an indication whena sentence is easy and when it is in need of guidance from acomputationally expensive model like the LSTM model 244. A threshold isapplied to the probability of the BoW model 112. LSTM model 244 isapplied on all sentences for which the probability of sentimentclassification is less than the set threshold. Therefore, the BoW model112 is used for linguistically easy sentences and the LSTM model 244 isused for linguistically complex sentences.

FIG. 8B and FIG. 8D are the data amount plots 800B and 800Drespectively. The height of bars on these two plots show the amount ofdata within the two thresholds and the line on the top is accumulateddata from each threshold bin.

System Overview—Deep Neural Network-Based Decision Network

We describe a system for a deep neural network-based decision network toselect on an input-by-input basis, a neural network-based model (alsoreferred to as a neural network-based classifier) to perform a machineclassification task. The neural network-based model is selected from apool of candidate neural network-based models (also referred to asneural network-based classifiers). The system and processes aredescribed with reference to FIG. 9 showing an architectural levelschematic of a system in accordance with an implementation. Because FIG.9 is an architectural diagram, certain details are intentionally omittedto improve the clarity of the description.

The discussion of FIG. 9 is organized as follows. First, the elements ofthe figure are described, followed by their interconnections. Then, theuse of the elements in the system is described in greater detail.

FIG. 9 includes the system 900. The system 900 includes machine learningsystem 905, a network 120, a training database 142, a validationdatabase 144, a splitter 910, a label transformer 920, and a modelselector 930. The machine learning system 905 includes a decisionnetwork 902, the bag of words (abbreviated BoW) model 112, and therecurrent neural network (abbreviated RNN) model 114. The trainingdatabase 142 and the validation database 144 contain sentiment analysisdata such as natural language sentences with positive and negativesentiment labels for each sentence. In other implementations, the system100 also includes a testing database with testing data that is used totest the machine learning system 905 during inference (production ortesting).

The decision network 902 is a neural network-based classifier thatoutputs probabilities to select either the BoW model 112 or the RNNmodel 114 to perform the sentiment classification task on the inputsentences. The machine learning system 905, the training database 142,the validation database 144 and the splitter 910, the label transformer920, and the model selector 930 are in communication with each other viathe network(s) 120. After presenting a high level description of thesystem 900, the discussion now turns to detailed description of variouscomponents of the system 900.

Data Preparation

FIG. 10 shows the operation 1000 of the splitter 910 to create atraining database 1030 and a validation database 1040. The trainingdatabase 1030 and the validation database 1040 can be part of the singletraining database 142. In one implementation, the splitter 910 createsthe training database 1030 and the validation database 1040 by assigning80% of the data from the training database 142 to the training database1030 and 20% of the training data from the training database 142 to thevalidation database 1040. In other implementations, the splitter 910 canassign the data to the training database 1030 and the validationdatabase 1040 in different proportions than described above.

Model Training and Inference—Sentiment Analysis Task

FIG. 11 illustrates training 1100 of the neural-network based models inthe candidate pool of neural network models to perform theclassification task using a training set. In one implementation, theclassification task can be sentiment classification of a sentence. Thesentiment classification task can comprise classifying a sentence ashaving a positive or negative sentiment. In other implementations, thesentiment classification task can comprise classifying the sentence ashaving a very positive, somewhat positive, neutral, very negative, orsomewhat negative sentiment. In yet other implementations, additional,fewer, or different class labels can be used. In furtherimplementations, other classification tasks can be used for trainingsuch as speech recognition, machine translation, text summarization,question answering, image captioning, and text-to-speech (TTS)synthesis.

The training database 1030 contains sentences with sentiment labels thatare used to separately train the LSTM model 244 and the BoW model 112.In one implementation, the LSTM model 244 and the BoW model 112 areseparately trained using the training database 1030 that contains 80% ofthe training data from the training database 142 (e.g., usingbackpropagation). The models are trained on the machine classificationtask. In one implementation, the machine classification task issentiment classification and the inputs are sentences. Each inputsentence in the training data 1030 can have a task label that indicateswhether the sentence has a positive sentiment or a negative sentiment.In one implementation, the resulting trained models are referred toherein as LSTM^(80%) 1110 and BoW^(80%) 1120 because they are trained on80% of the training data.

The trained models LSTM^(80%) 1110 and BoW^(80%) 1120 are tested withvalidation data 1040 separately. The validation data 1040 contains aportion of the data from the training database 142 that was not used totrain and produce the LSTM^(80%) 1110 and BoW^(80%) 1120. In oneimplementation, the validation data contains 20% of the data from thetraining database 142. As shown in FIG. 12, the trained modelsLSTM^(80%) 1110 and BoW^(80%) 1120 generate sentiment class labels 1200for each input sentence. The inferred sentiment class labels (eitherpositive or negative) 1200 for each input sentence are compared with therespective ground truth sentiment class labels.

Confusion Matrix

The results of comparison of inferred sentiment class labels with groundtruth sentiment class labels are used to generate 1300 a “confusionmatrix” 1310, as shown in FIG. 13. Confusion matrix is a metric foraccurately evaluating classification models. For a classificationproblem with C classes, the confusion matrix M is a C×C matrix whereelements M_(ij) in this matrix show the percentage of samples whoseactual class labels are i but they are classified under class j usingthe neural network classifier. M_(ij) shows the percentage of sampleswhich are correctly classified. For binary classification problems inwhich there are two classes, the confusion matrix is a 2×2 matrix 1310as shown in FIG. 13.

Element M₁₁ 1312 of the confusion matrix 1310 shows the percentage ofsamples whose actual labels are “true” and they are correctly classifiedas “true” during prediction. Element M₁₁ 1312 comprises 76%true-positive (TP) samples. Element M₁₂ 1314 of the confusion matrix1310 shows the percentage of samples whose actual label is “true” butthey are incorrectly classified as “false” during prediction. ElementM₁₂ 1314 comprises 12% false-negative (FN) samples. Element M₂₁ 1316 ofthe confusion matrix 1310 shows the percentage of samples whose actuallabel is “false” but they are incorrectly classified as “true” duringprediction. Element M₂₁ 1316 comprises 6% false-positive (FP) samples.Element M₂₂ 1318 of the confusion matrix 1310 shows the percentage ofsamples whose actual label is “false” and they are correctly classifiedas “false” during prediction. Element M₂₂ 1318 comprises 6%true-negative (TN) samples.

Classification results generated by the trained models LSTM^(80%) 1110and BoW^(80%) 1120 on validation data 1040 (FIG. 12) can be plotted intwo separate confusion matrices, according to one implementation. Theresults from both confusion matrices for the two classifiers, LSTM^(80%)1110 and BoW^(80%) 1120, can also be combined in a single confusionmatrix 1310. The resulting confusion matrix 1310 identifies:

-   -   Inputs in validation database 1040 accurately classified by both        the trained recurrent neural network-based classifier LSTM^(80%)        1110 and the trained non-recurrent neural network-based        classifier BoW^(80%) 1120 (true-positive, top left). This means        that the LSTM and the BoW models are both correct 76% of the        time.    -   Inputs in validation database 1040 inaccurately classified by        both the trained recurrent neural network-based classifier        LSTM^(80%) 1110 and the trained non-recurrent neural        network-based classifier BoW^(80%) 1120 (true-negative, bottom        right). This means that the LSTM and the BoW models are both        incorrect 6% of the time.    -   Inputs in validation database 1040 accurately classified by the        trained non-recurrent neural network-based classifier BoW^(80%)        1120 but inaccurately classified by the trained recurrent neural        network-based classifier LSTM^(80%) 1110 (false-positive, bottom        left). This means that, when the LSTM model is incorrect, the        BoW model is correct 6% of the time.    -   Inputs in validation database 1040 accurately classified by the        trained recurrent neural network-based classifier LSTM^(80%)        1110 but inaccurately classified by the trained non-recurrent        neural network-based classifier BoW^(80%) 1120 (false-negative,        top right). This means that, when the BoW model is incorrect,        the LSTM model is correct 12% of the time.

The label transformer 920 assigns new labels to inputs in the validationdatabase 1040 and generates a new validation database 1340. In oneimplementation, the label transformer 920 groups the inputs invalidation database 1040 into two subsets in the new validation database1340. A first subset of inputs is assigned a first model class labelidentifying the trained recurrent neural network-based classifierLSTM^(80%) 1110. A second subset of inputs is assigned a second modelclass label identifying the trained non-recurrent neural network-basedclassifier BoW^(80%) 1120.

The first subset of inputs in the validation data 1340 includes inputsin validation database 1040 accurately classified by the trainedrecurrent neural network-based classifier LSTM^(80%) 1110 butinaccurately classified by the trained non-recurrent neuralnetwork-based classifier BoW^(80%) 1120. The second subset of inputs inthe validation data 1340 includes the inputs in validation database 1040in remaining three categories from the confusion matrix 1310. Theseinputs include: inputs in validation database 1040 accurately classifiedby both the trained recurrent neural network-based classifier LSTM^(80%)1110 and the trained non-recurrent neural network-based classifierBoW^(80%) 1120, inputs in validation database 1040 inaccuratelyclassified by both the trained recurrent neural network-based classifierLSTM^(80%) 1110 and the trained non-recurrent neural network-basedclassifier BoW^(80%) 1120, and inputs in validation database 1040accurately classified by the trained non-recurrent neural network-basedclassifier BoW^(80%) 1120 but inaccurately classified by the trainedrecurrent neural network-based classifier LSTM^(80%) 1110. The new modelclass labeled input data is stored in the validation database 1340. Thedata in the validation database 1340 is also referred to as a decisionset. In another implementation, the label transformer 920 adds modelclass labels to inputs in the existing validation database 1040. Thisresults in the inputs having two labels each—a task class label,identifying a positive or a negative sentiment and a model class label,identifying a trained recurrent neural network-based classifierLSTM^(80%) 1110 or a trained non-recurrent neural network-basedclassifier BoW^(80%) 1120.

In some implementations, the original validation data is not replicatedto form the new validation data and instead each data item in thevalidation data is assigned and stored with two ground truth pointers;one for the sentiment class label and one for the model class label. Inyet other implementations, two copies of the validation data aremaintained with separate and different ground truth labels (one for thesentiment class label and one for the model class label).

Decision Network

FIG. 14 shows training 1400 of the decision network (also referred toherein as decision classifier) 902 using inputs with model class labelsin the validation database 1340. The outputs of the decision network 902are inferred model class labels for each input sentence in thevalidation database 1340. The inferred model class label for eachsentence by the decision network 902 is compared to ground truth modelclass label for that sentence in the validation data 1340.

FIG. 15 illustrates one implementation of architecture 1500 of thedecision network 902. In other implementations, any other conventionalor future-developed BoW, RNN, or LSTM architectures can be used herein.Such implementations will be readily apparent to those skilled in theart without departing from the spirit and scope of the technologydisclosed.

In one implementation, the decision network 902 comprises the trainednon-recurrent neural network-based classifier BoW^(80%) 1120 with theoriginal classification layer removed from the architecture as shown inFIG. 902. One implementation of the architecture of the original BoWmodel 112 is shown in FIG. 4. The new classification layer in thedecision network 902 classifies in the input sentences using the modelclass labels and produces output probabilities for the first and secondmodel class labels. In the implementation described above, theseprobabilities correspond to LSTM^(80%) 1110 and BoW^(80%) 1120.Additionally, two fully-connected layers with ReLU are included in thedecision network 902.

In some implementations, during training of the decision network 902,gradients only for the fully-connected layers and the new classificationlayer are backpropagated. The weights of the trained non-recurrentneural network-based classifier BoW^(80%) 1120 are kept fixed and remainunchanged.

Process Flow

FIG. 16 is a flowchart 1600 illustrating process steps for modelselection using decision network of FIG. 9. The process starts at step1610 when a sentence is given as input to the decision network 902. Thedecision network 902 then generates a model classification at step 1612.At step 1614, either the LSTM model or the BoW model is selected basedon the classification by the decision network 902 at step 1612.

If the LSTM model is selected at step 1614, then the sentence isprovided to the LSTM^(100%) model for sentiment classification.LSTM^(100%) is trained on the entire training data 142, which in turncomprises the training data 1030 and the validation data 1040. At step1616, the LSTM^(100%) produces a sentiment classification for thesentence. At step 1618, sentiment classification of the LSTM^(100%) isaccepted.

If the BoW model is selected at step 1614, then the sentence is providedto the BoW^(100%) m for sentiment classification. BoW^(100%) is trainedon the entire training data 142, which in turn comprises the trainingdata 1030 and the validation data 1040. At step 1622, the BoW^(100%)produces a sentiment classification for the sentence. At step 1624,sentiment classification of the BoW^(100%) is accepted.

At step 1620, the system checks if there are more input sentences forsentiment classification. If this is true, then the next sentence isgiven as input to the decision network 902 and the process is repeated.The process ends at step 1626 when there are no further input sentencesfor sentiment classification.

Computer System

FIG. 17 is a simplified block diagram of a computer system 1700 that canbe used to implement the machine learning systems 110 and 905 of FIG. 1and FIG. 9. Computer system 1700 includes at least one centralprocessing unit (CPU) 1724 that communicates with a number of peripheraldevices via bus subsystem 1722. These peripheral devices can include astorage subsystem 1710 including, for example, memory devices and a filestorage subsystem 1718, user interface input devices 1720, userinterface output devices 1728, and a network interface subsystem 1726.The input and output devices allow user interaction with computer system1700. Network interface subsystem 1726 provides an interface to outsidenetworks, including an interface to corresponding interface devices inother computer systems.

In one implementation, the machine learning system 110 of FIG. 1 and themachine learning system 905 of FIG. 9 are communicably linked to thestorage subsystem 1710 and the user interface input devices 1720.

User interface input devices 1720 can include a keyboard; pointingdevices such as a mouse, trackball, touchpad, or graphics tablet; ascanner; a touch screen incorporated into the display; audio inputdevices such as voice recognition systems and microphones; and othertypes of input devices. In general, use of the term “input device” isintended to include all possible types of devices and ways to inputinformation into computer system 1700.

User interface output devices 1728 can include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem can include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem can also provide a non-visual display such as audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computer system 1700 to the user or to another machine or computersystem.

Storage subsystem 1710 stores programming and data constructs thatprovide the functionality of some or all of the modules and methodsdescribed herein. These software modules are generally executed by deeplearning processors 1730.

Deep learning processors 1730 can be graphics processing units (GPUs) orfield-programmable gate arrays (FPGAs). Deep learning processors 1730can be hosted by a deep learning cloud platform such as Google CloudPlatform™, Xilinx™, and Cirrascale™. Examples of deep learningprocessors 1730 include Google's Tensor Processing Unit (TPU)™,rackmount solutions like GX4 Rackmount Series™, GX8 Rackmount Series™,NVIDIA DGX-1™, Microsoft′ Stratix V FPGA™, Graphcore's IntelligentProcessor Unit (IPU)™, Qualcomm's Zeroth Platform™ with SnapdragonProcessors™, NVIDIA's Volta™, NVIDIA's DRIVE PX™, NVIDIA's JETSONTX1/TX2 MODULE™, Intel's Nirvana™, Movidius VPU™, Fujitsu DPI™, ARM'sDynamicIQ™, IBM TrueNorth™, and others.

Memory subsystem 1712 used in the storage subsystem 1710 can include anumber of memories including a main random access memory (RAM) 1714 forstorage of instructions and data during program execution and a readonly memory (ROM) 1716 in which fixed instructions are stored. A filestorage subsystem 1718 can provide persistent storage for program anddata files, and can include a hard disk drive, a floppy disk drive alongwith associated removable media, a CD-ROM drive, an optical drive, orremovable media cartridges. The modules implementing the functionalityof certain implementations can be stored by file storage subsystem 1718in the storage subsystem 1710, or in other machines accessible by theprocessor.

Bus subsystem 1722 provides a mechanism for letting the variouscomponents and subsystems of computer system 1700 communicate with eachother as intended. Although bus subsystem 1722 is shown schematically asa single bus, alternative implementations of the bus subsystem can usemultiple busses.

Computer system 1700 itself can be of varying types including a personalcomputer, a portable computer, a workstation, a computer terminal, anetwork computer, a television, a mainframe, a server farm, awidely-distributed set of loosely networked computers, or any other dataprocessing system or user device. Due to the ever-changing nature ofcomputers and networks, the description of computer system 1700 depictedin FIG. 17 is intended only as a specific example for purposes ofillustrating the preferred embodiments of the present invention. Manyother configurations of computer system 1700 are possible having more orless components than the computer system depicted in FIG. 17.

The preceding description is presented to enable the making and use ofthe technology disclosed. Various modifications to the disclosedimplementations will be apparent, and the general principles definedherein may be applied to other implementations and applications withoutdeparting from the spirit and scope of the technology disclosed. Thus,the technology disclosed is not intended to be limited to theimplementations shown, but is to be accorded the widest scope consistentwith the principles and features disclosed herein. The scope of thetechnology disclosed is defined by the appended claims.

Particular Implementations

Probability-Based Guider

The technology disclosed relates to efficient performance of a machineclassification task on a dataset comprising a plurality of inputs. Inone implementation, the machine classification task can be sentimentclassification of a sentence. The sentiment classification task cancomprise classifying a sentence as having a positive or negativesentiment. In other implementations, the sentiment classification taskcan comprise classifying the sentence as having a very positive,somewhat positive, neutral, very negative, or somewhat negativesentiment. In yet other implementations, additional, fewer, or differentclass labels can be used. In further implementations, other machineclassification tasks can be performed such as speech recognition,machine translation, text summarization, question answering, imagecaptioning, and text-to-speech (TTS) synthesis.

The technology disclosed can be practiced as a system, method, orarticle of manufacture. One or more features of an implementation can becombined with the base implementation. Implementations that are notmutually exclusive are taught to be combinable. One or more features ofan implementation can be combined with other implementations. Thisdisclosure periodically reminds the user of these options. Omission fromsome implementations of recitations that repeat these options should notbe taken as limiting the combinations taught in the precedingsections—these recitations are hereby incorporated forward by referenceinto each of the following implementations.

A first system implementation of the technology disclosed includes oneor more processors coupled to the memory. The memory is loaded withcomputer instructions to efficiently perform a machine classificationtask on a dataset comprising a plurality of input. The system processesthe input through a first non-recurrent neural network which performsthe machine classification task on the inputs. The first non-recurrentneural network generates a mean vector representation of an input byaveraging vector embeddings of the input. The system then processes aselected subset of inputs through a second recurrent neural network(abbreviated RNN) to perform the machine classification task on theinputs in the subset. The selection for further processing isconditioned on the first non-recurrent neural network's confidence inperforming the machine classification task on the inputs in the dataset.

The first system implementation and other systems disclosed optionallyinclude one or more of the following features. System can also includefeatures described in connection with methods disclosed. In the interestof conciseness, alternative combinations of system features are notindividually enumerated. Features applicable to systems, methods, andarticles of manufacture are not repeated for each statutory class set ofbase features. The reader will understand how features identified inthis section can readily be combined with base features in otherstatutory classes.

The second RNN included in the system is at least three percent moreaccurate and four times computationally more expensive than the firstnon-recurrent neural network. The system selects the second RNN when thefirst non-recurrent neural network's confidence is below a setthreshold. The machine classification task is to classify an input to afirst class or a second class. The first non-recurrent neural network'sconfidence represents a probability score assigned to either the firstclass or the second class. The input is a sentence and the machineclassification task is to classify the sentence to a positive sentimentclass or a negative sentiment class.

The first non-recurrent neural network included in the system produces aprobability score for a linguistically complex sentence which is atleast twenty percent lower than a class probability produced by thefirst non-recurrent neural network for a linguistically simple sentence.

The system includes a second RNN which is used to classify alinguistically complex sentence to the positive sentiment class or thenegative sentiment class. In one implementation, the first non-recurrentneural network included in the system is a bag of words (abbreviatedBoW) network. In another implementation the first non-recurrent neuralnetwork is at least one continuous bag of words (abbreviated CBoW)network. In another implementation, the first non-recurrent neuralnetwork is at least one skip-gram network. In yet anotherimplementation, the first non-recurrent neural network is at least oneconvolutional neural network (abbreviated CNN). In anotherimplementation, the second RNN is a long short-term memory (abbreviatedLSTM) network. In another implementation, the second RNN is a gatedrecurrent unit (abbreviated GRU) network. In yet another implementation,the second RNN is a quasi-recurrent neural network (abbreviated QRNN).

In one implementation, the system includes a threshold that is a singlevalue. In another implementation, the system includes a threshold thatis a range between two values.

In one implementation, the machine classification task is apart-of-speech (abbreviated POS) tagging. In another implementation, themachine classification task is chunking. In another implementation, themachine classification task is dependency parsing. In anotherimplementation, the machine classification task is semantic relatedness.In yet another implementation, the machine classification task istextual entailment.

Examples of some machine classification tasks that can be performed bythe technology disclosed include speech recognition, machinetranslation, text summarization, question answering, image captioning,and text-to-speech (TTS) synthesis.

A second system implementation of the technology disclosed includes oneor more processors coupled to the memory. The memory is loaded withcomputer instructions to efficiently perform a sentiment classificationtask on a sentence. The system comprises a first non-recurrent neuralnetwork that evaluates the sentence and produces a confidence score.This score specifies a likelihood of the sentence's sentiment beingpositive or negative. The first non-recurrent neural network generates amean vector representation of an input by averaging vector embeddings ofthe input. The system further comprises of a guider that compares theconfidence score produced by the first non-recurrent neural networkagainst a set threshold. Based on this comparison the guider determines,whether the sentence requires supplemental evaluation by a secondrecurrent neural network (abbreviated RNN). The supplemental evaluationincludes when the confidence score is below the threshold, using thesecond RNN to classify the sentence's sentiment as positive or negative.When the confidence score is above the threshold, relying on theconfidence score produced by the first non-recurrent neural network forthe sentiment classification task without requiring the supplementalevaluation by the second RNN.

The second system implementation and other systems disclosed optionallyinclude one or more of the following features. System can also includefeatures described in connection with methods disclosed. In the interestof conciseness, alternative combinations of system features are notindividually enumerated. Features applicable to systems, methods, andarticles of manufacture are not repeated for each statutory class set ofbase features. The reader will understand how features identified inthis section can readily be combined with base features in otherstatutory classes.

The first non-recurrent neural network in the system produces a twentypercent lower confidence score when the sentence is a linguisticallycomplex sentence compared to when the sentence is a linguisticallysimple sentence. In such an implementation, the linguistically complexsentence has co-occurrences of negative and positive constituents. Inanother such implementation, the linguistically complex sentence hasmultiple instances of negative words and contrastive conjunctions. Inanother such implementation, the system includes using the second RNN toclassify the linguistically complex sentence's sentiment as positive ornegative.

Each of the features discussed in this particular implementation sectionfor the first system implementation apply equally to the second systemimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by a processor to performfunctions of the system described above. Yet another implementation mayinclude a method performing the functions of the system described above.

A third system implementation of the technology disclosed includes morethan one processors operating in parallel and coupled to memory. Thememory is loaded with computer instructions to efficiently perform amachine classification task on a dataset comprising a plurality ofinputs. The instructions upon execution on the processors implement theactions comprising processing the inputs through a first non-recurrentneural network which performs the machine classification task on theinputs. The processing by the first non-recurrent neural networkgenerates a mean vector representation of an input by averaging vectorembeddings of the input.

Following this, the system processes a selected subset of inputs througha second recurrent neural network (abbreviated RNN) to perform themachine classification task on the inputs in the subset. The selectionof the inputs for further processing is conditioned on the firstnon-recurrent neural network's confidence in performing the machineclassification task on the inputs in the dataset.

Each of the features discussed in this particular implementation sectionfor the first and the second system implementation apply equally to thethird system implementation. As indicated above, all the system featuresare not repeated here and should be considered repeated by reference.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by a processor to performfunctions of the system described above. Yet another implementation mayinclude a method performing the functions of the system described above.

A first method implementation of the technology disclosed includesefficiently performing a machine classification task on a datasetcomprising a plurality of inputs. The method includes, processing theinputs through a first non-recurrent neural network which performs themachine classification task on the inputs. The first non-recurrentneural network generates a mean vector representation of an input byaveraging vector embeddings of the input. The method then processes aselected subset of inputs through a second recurrent neural network(abbreviated RNN) to perform the machine classification task on theinputs in the subset. The selection for further processing isconditioned on the first non-recurrent neural network's confidence inperforming the machine classification task on the inputs in the dataset.

Each of the features discussed in this particular implementation sectionfor the first system implementation apply equally to this methodimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by a processor to performthe first method described above. Yet another implementation may includea system including memory and one or more processors operable to executeinstructions, stored in the memory, to perform the first methoddescribed above.

Computer readable media (CRM) implementations of the technologydisclosed include a non-transitory computer readable storage mediumimpressed with computer program instructions, when executed on aprocessor, implement the method described above.

Each of the features discussed in this particular implementation sectionfor the first system implementation apply equally to the CRMimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

A second method implementation of the technology disclosed includesefficiently perform a sentiment classification task on a sentence. Themethod includes evaluating a sentence using a first non-recurrent neuralnetwork and producing a confidence score which specifies a likelihood ofthe sentence's sentiment being positive or negative. The method includesgenerating a mean vector representation of an input by averaging vectorembeddings of the input by the first non-recurrent neural network.Following this, the method includes using a guider to compares theconfidence score produced by the first non-recurrent neural networkagainst a set threshold. Based on this comparison, the method includes,determining whether the sentence requires supplemental evaluation by asecond recurrent neural network (abbreviated RNN). The method furtherincludes, based on this evaluation, using the second RNN to classify thesentence's sentiment as positive or negative when the confidence scoreis below the threshold. The method includes relying on the confidencescore produced by the first non-recurrent neural network for thesentiment classification task, without requiring the supplementalevaluation by the second RNN when the confidence score is above thethreshold.

Each of the features discussed in this particular implementation sectionfor the second system implementation apply equally to this methodimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by a processor to performthe second method described above. Yet another implementation mayinclude a system including memory and one or more processors operable toexecute instructions, stored in the memory, to perform the second methoddescribed above.

Computer readable media (CRM) implementations of the technologydisclosed include a non-transitory computer readable storage mediumimpressed with computer program instructions, when executed on aprocessor, implement the method described above.

Each of the features discussed in this particular implementation sectionfor the second system implementation apply equally to the CRMimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

A third method implementation of the technology disclosed includesefficiently perform a machine classification task on a datasetcomprising a plurality of inputs. The method further includes processingthe inputs through a first non-recurrent neural network which performsthe machine classification task on the inputs, wherein the firstnon-recurrent neural network generates a mean vector representation ofan input by averaging vector embeddings of the input. Following this,the method includes further processing a selected subset of inputsthrough a second recurrent neural network (abbreviated RNN) to performthe machine classification task on the inputs in the subset. Theselection for further processing is conditioned on the firstnon-recurrent neural network's confidence in performing the machineclassification task on the inputs in the dataset.

Each of the features discussed in this particular implementation sectionfor the third system implementation apply equally to this methodimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by a processor to performthe third method described above. Yet another implementation may includea system including memory and one or more processors operable to executeinstructions, stored in the memory, to perform the third methoddescribed above.

Computer readable media (CRM) implementations of the technologydisclosed include a non-transitory computer readable storage mediumimpressed with computer program instructions, when executed on aprocessor, implement the method described above.

Each of the features discussed in this particular implementation sectionfor the third system implementation apply equally to the CRMimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

Deep Neural Network-Based Decision Network

The technology disclosed relates to efficient performance of a machineclassification task on a dataset comprising a plurality of inputs.

The technology disclosed can be practiced as a system, method, orarticle of manufacture. One or more features of an implementation can becombined with the base implementation. Implementations that are notmutually exclusive are taught to be combinable. One or more features ofan implementation can be combined with other implementations. Thisdisclosure periodically reminds the user of these options. Omission fromsome implementations of recitations that repeat these options should notbe taken as limiting the combinations taught in the precedingsections—these recitations are hereby incorporated forward by referenceinto each of the following implementations.

A first system implementation of the technology disclosed includes oneor more processors coupled to the memory. The memory is loaded withcomputer instructions to select on an input-by-input basis, a neuralnetwork-based classifier from a pool of candidate neural network-basedclassifiers to perform a machine classification task in a manner thatmaximizes accuracy and minimizes computational cost. The system performsthe training of the candidate neural network-based classifiers in thepool, to perform the machine classification task, using a training set.The training set comprises training inputs annotated with task classlabels defined for the machine classification task. The system then usesthe trained candidate neural network-based classifiers to perform themachine classification task on a validation set. The validation set iscomprised of validation inputs annotated with the task class labels. Thesystem then generates a confusion matrix based on the trained neuralnetwork-based classifiers' performance. The system then uses theconfusion matrix to identify a first subset of validation inputsaccurately inferred only by a particular trained recurrent neuralnetwork-based classifier in the pool and a second subset of validationinputs comprising validation inputs not in the first subset. The systemthen labels the validation inputs in the first subset with a first modelclass label identifying the particular trained recurrent neuralnetwork-based classifier. The system labels the validation inputs in thesecond subset with a second model class label identifying a particulartrained non-recurrent neural network-based classifier in the pool. Thesystem stores the model-class-labeled validation inputs in the first andsecond subsets as a decision set.

The first system implementation and other systems disclosed optionallyinclude one or more of the following features. System can also includefeatures described in connection with methods disclosed. In the interestof conciseness, alternative combinations of system features are notindividually enumerated. Features applicable to systems, methods, andarticles of manufacture are not repeated for each statutory class set ofbase features. The reader will understand how features identified inthis section can readily be combined with base features in otherstatutory classes.

The system performs the machine classification task for a given inputduring inference, based on output probabilities of the trained neuralnetwork-based decision classifier using either the particular trainedrecurrent neural network-based classifier or the particular trainednon-recurrent neural network-based classifier. The system selects theparticular trained recurrent neural network-based classifier when theoutput probability of the first model class label is higher than that ofthe second model class label. The system selects the particular trainednon-recurrent neural network-based classifier when the outputprobability of the second model class label is higher than that of thefirst model class label.

In one implementation of the system, the particular trained recurrentneural network-based classifier is at least three percent more accurateand four times computationally more expensive than the particulartrained non-recurrent neural network-based classifier. In one suchimplementation, the particular trained recurrent neural network-basedclassifier is at least one recurrent neural network (abbreviated RNN).In another implementation, the particular trained non-recurrent neuralnetwork-based classifier is at least one bag of words (abbreviated BoW)network. In another such implementation, the particular trainednon-recurrent neural network-based classifier is at least one continuousbag of words (abbreviated CBoW) network. In another implementation, theparticular trained non-recurrent neural network-based classifier is atleast one skip-gram network. In one implementation, the particulartrained non-recurrent neural network-based classifier is at least oneconvolutional neural network (abbreviated CNN). In one implementation,the RNN is a long short-term memory (abbreviated LSTM) network. Inanother implementation, the RNN is a gated recurrent unit (abbreviatedGRU) network. In yet another implementation, the RNN is aquasi-recurrent neural network (abbreviated QRNN).

In one implementation of the system, the training set and the validationset are part of a single data set that is subjected to held-outsplitting to create the training set and the validation set. Each of thecandidate neural network-based classifiers in the pool is trainedseparately on the training set. The machine classification task issentiment classification and the inputs are sentence. The task classlabels are positive sentiment and negative sentiment.

The confusion matrix identifies at least one of (1) validation inputsaccurately classified by both the particular trained recurrent neuralnetwork-based classifier and the particular trained non-recurrent neuralnetwork-based classifier, (2) validation inputs inaccurately classifiedby both the particular trained recurrent neural network-based classifierand the particular trained non-recurrent neural network-basedclassifier, (3) validation inputs accurately classified by theparticular trained non-recurrent neural network-based classifier butinaccurately classified by the particular trained recurrent neuralnetwork-based classifier, and (4) validation inputs accuratelyclassified by the particular trained recurrent neural network-basedclassifier but inaccurately classified by the particular trainednon-recurrent neural network-based classifier.

In such an implementation of the system, first subset includesvalidation inputs accurately classified by the particular trainedrecurrent neural network-based classifier but inaccurately classified bythe particular trained non-recurrent neural network-based classifier.The second subset includes at least one of: validation inputs accuratelyclassified by both the particular trained recurrent neural network-basedclassifier and the particular trained non-recurrent neural network-basedclassifier; validation inputs inaccurately classified by both theparticular trained recurrent neural network-based classifier and theparticular trained non-recurrent neural network-based classifier; andvalidation inputs accurately classified by the particular trainednon-recurrent neural network-based classifier but inaccuratelyclassified by the particular trained recurrent neural network-basedclassifier.

The particular trained recurrent neural network-based classifier and theparticular trained non-recurrent neural network-based classifierselected by the trained neural network-based decision classifier duringinference are trained on a combination of the training set and thevalidation set, e trained LSTM^(100%) and BoW^(100%). The neuralnetwork-based decision classifier comprises the particular trainednon-recurrent neural network-based classifier trained on the trainingset, with an original classification layer ablated. It includes one ormore new fully-connected layers, and a new classification layer thatproduces output probabilities for the first and second model classlabels. During training of the neural network-based decision classifier,only gradients for the fully-connected layers and the new classificationlayer are backpropagated and weights of the particular trainednon-recurrent neural network-based classifier are kept fixed. In oneimplementation, the particular trained non-recurrent neuralnetwork-based classifier is at least one bag of words (abbreviated BoW)network.

Examples of some machine classification tasks that can be performed bythe technology disclosed include speech recognition, machinetranslation, text summarization, question answering, image captioning,and text-to-speech (TTS) synthesis.

A second system implementation of the technology disclosed includes aneural network-based decision system with processors operating inparallel. The neural network-based decision system efficiently performsa sentiment classification task on a sentence using either a trainedrecurrent long short-term memory (abbreviated LSTM) network or a trainedbag of words (abbreviated BoW) network that generates a mean vectorrepresentation of a sentence by averaging token vector embeddings of thesentence. The system generates a confusion matrix based on the trainedLSTM and BoW networks' evaluation of validation sentences annotated withpositive and negative sentiment labels. The system uses the confusionmatrix to identify a subset of validation sentences accuratelyclassified only by the trained LSTM network. The system then annotatesthe subset of validation sentences with a first model label identifyingthe trained LSTM network and annotates remaining of the validationsentences with a second model label identifying the trained BoW network.The system stores the model-annotated validation sentences in a decisionset. The system constructs a decision system using the trained BoWnetwork and trains the decision system using the decision set to producean output that specifies whether to use the trained LSTM network or thetrained BoW network for classifying the sentence's sentiment.

The second system implementation and other systems disclosed optionallyinclude one or more of the following features. System can also includefeatures described in connection with methods disclosed. In the interestof conciseness, alternative combinations of system features are notindividually enumerated. Features applicable to systems, methods, andarticles of manufacture are not repeated for each statutory class set ofbase features. The reader will understand how features identified inthis section can readily be combined with base features in otherstatutory classes.

During inference, the system uses an output of the trained decisionsystem to select either the trained LSTM network or the trained BoWnetwork to classify a sentence's sentiment. The decision system selectsthe trained LSTM network when the sentence is a linguistically complexsentence. The decision system selects the trained BoW network when thesentence is a linguistically simple sentence.

Each of the features discussed in this particular implementation sectionfor the first system implementation apply equally to the second systemimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by a processor to performfunctions of the system described above. Yet another implementation mayinclude a method performing the functions of the system described above.

A third system implementation of the technology disclosed includes asystem with processors operating in parallel. The system comprises athird neural network-based classifier that selects, on an input-by-inputbasis, between first and second neural network-based classifiers trainedto perform a classification task. The selection is governed by the thirdneural network-based classifier's training on a training set annotatedwith class labels that distinguish between inputs accurately classifiedonly by the trained second neural network-based classifier and remaininginputs in the training set.

The third system implementation and other systems disclosed optionallyinclude one or more of the following features. System can also includefeatures described in connection with methods disclosed. In the interestof conciseness, alternative combinations of system features are notindividually enumerated. Features applicable to systems, methods, andarticles of manufacture are not repeated for each statutory class set ofbase features. The reader will understand how features identified inthis section can readily be combined with base features in otherstatutory classes.

The remaining inputs include inputs inaccurately classified by thetrained second neural network-based classifier and inputs accuratelyclassified by both the trained first and second neural network-basedclassifiers.

Each of the features discussed in this particular implementation sectionfor the first and the second system implementations apply equally to thethird system implementation. As indicated above, all the system featuresare not repeated here and should be considered repeated by reference.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by a processor to performfunctions of the system described above. Yet another implementation mayinclude a method performing the functions of the system described above.

A first method implementation of the technology disclosed includesselecting, on an input-by-input basis, a neural network-based classifierfrom a pool of candidate neural network-based classifiers to perform amachine classification task in a manner that maximizes accuracy andminimizes computational cost. The method includes, training thecandidate neural network-based classifiers in the pool to perform themachine classification task using a training set, the training setcomprising training inputs annotated with task class labels defined forthe machine classification task. Following this, the method includesusing the trained candidate neural network-based classifiers to performthe machine classification task on a validation set, the validation setcomprises validation inputs annotated with the task class labels. Themethod includes generating a confusion matrix based on the trainedneural network-based classifiers' performance. The confusion matrix isused to identify a first subset of validation inputs accurately inferredonly by a particular trained recurrent neural network-based classifierin the pool and a second subset of validation inputs comprisingvalidation inputs not in the first subset. Following this, the methodincludes labeling the validation inputs in the first subset with a firstmodel class label identifying the particular trained recurrent neuralnetwork-based classifier. The method also includes labeling thevalidation inputs in the second subset with a second model class labelidentifying a particular trained non-recurrent neural network-basedclassifier in the pool. Following this, the method includes, storing themodel-class-labeled validation inputs in the first and second subsets asa decision set. Finally, the method includes, training a neuralnetwork-based decision classifier using the decision set to outputprobabilities for the first and second model class labels on aninput-by-input basis. The output probabilities specify respectivelikelihoods of selecting the particular trained recurrent neuralnetwork-based classifier and the particular trained non-recurrent neuralnetwork-based classifier.

Each of the features discussed in this particular implementation sectionfor the first system implementation apply equally to this methodimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by a processor to performthe first method described above. Yet another implementation may includea system including memory and one or more processors operable to executeinstructions, stored in the memory, to perform the first methoddescribed above.

Computer readable media (CRM) implementations of the technologydisclosed include a non-transitory computer readable storage mediumimpressed with computer program instructions, when executed on aprocessor, implement the method described above.

Each of the features discussed in this particular implementation sectionfor the first system implementation apply equally to the CRMimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

A second method implementation of the technology disclosed includesefficiently performing a sentiment classification task on a sentenceusing either a trained recurrent long short-term memory (abbreviatedLSTM) network or a trained bag of words (abbreviated BoW) network thatgenerates a mean vector representation of a sentence by averaging tokenvector embeddings of the sentence. The method includes generating aconfusion matrix based on the trained LSTM and BoW networks' evaluationof validation sentences annotated with positive and negative sentimentlabels. Following this, the method includes, using the confusion matrixto identify a subset of validation sentences accurately classified onlyby the trained LSTM network and annotating the subset of validationsentences with a first model label identifying the trained LSTM networkand annotating remaining of the validation sentences with a second modellabel identifying the trained BoW network. Then the method includes,storing the model-annotated validation sentences in a decision set.Finally, the method includes, constructing a decision system using thetrained BoW network and training the decision system using the decisionset to produce an output that specifies whether to use the trained LSTMnetwork or the trained BoW network for classifying the sentence'ssentiment.

Each of the features discussed in this particular implementation sectionfor the second system implementation apply equally to this methodimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by a processor to performthe second method described above. Yet another implementation mayinclude a system including memory and one or more processors operable toexecute instructions, stored in the memory, to perform the second methoddescribed above.

Computer readable media (CRM) implementations of the technologydisclosed include a non-transitory computer readable storage mediumimpressed with computer program instructions, when executed on aprocessor, implement the method described above.

Each of the features discussed in this particular implementation sectionfor the second system implementation apply equally to the CRMimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

A third method implementation of the technology disclosed includesselecting using a third neural network-based classifier, on aninput-by-input basis, between first and second neural network-basedclassifiers trained to perform a classification task. The selectiongoverned by the third neural network-based classifier's training on atraining set annotated with class labels that distinguish between inputsaccurately classified only by the trained second neural network-basedclassifier and remaining inputs in the training set.

Each of the features discussed in this particular implementation sectionfor the third system implementation apply equally to this methodimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by a processor to performthe third method described above. Yet another implementation may includea system including memory and one or more processors operable to executeinstructions, stored in the memory, to perform the third methoddescribed above.

Computer readable media (CRM) implementations of the technologydisclosed include a non-transitory computer readable storage mediumimpressed with computer program instructions, when executed on aprocessor, implement the method described above.

Each of the features discussed in this particular implementation sectionfor the third system implementation apply equally to the CRMimplementation. As indicated above, all the system features are notrepeated here and should be considered repeated by reference.

What is claimed is:
 1. A system with numerous processors operating inparallel, comprising: a decision neural network-based classifier thatselects, on an input-by-input basis, between non-recurrent and recurrentneural network-based classifiers trained to perform a machineclassification task, with the selection governed by the decision neuralnetwork-based classifier's training on a decision training set annotatedwith model class labels that distinguish between inputs accuratelyclassified only by the trained recurrent neural network-based classifierand remaining inputs in the decision training set.
 2. The system ofclaim 1, wherein the remaining inputs include inputs inaccuratelyclassified by the trained recurrent neural network-based classifier andinputs accurately classified by both the trained non-recurrent andrecurrent neural network-based classifiers.
 3. The system of claim 1,further configured to: train the non-recurrent and recurrent neuralnetwork-based classifiers to perform the machine classification taskusing a training set, the training set comprising training inputsannotated with task class labels defined for the machine classificationtask; use the trained non-recurrent and recurrent neural network-basedclassifiers to perform the machine classification task on a validationset, the validation set comprising validation inputs annotated with thetask class labels; generate a confusion matrix based on the trainednon-recurrent and recurrent neural network-based classifiers'performance and use the confusion matrix to identify a first subset ofvalidation inputs accurately inferred only by the trained recurrentneural network-based classifier and a second subset of validation inputscomprising validation inputs not in the first subset; label thevalidation inputs in the first subset with a first model class labelidentifying the trained recurrent neural network-based classifier, labelthe validation inputs in the second subset with a second model classlabel identifying the trained non-recurrent neural network-basedclassifier, and store the model-class-labeled validation inputs in thefirst and second subsets as the decision training set; and train thedecision neural network-based classifier using the decision training setto output probabilities for the first and second model class labels onan input-by-input basis, the output probabilities specifying respectivelikelihoods of selecting the trained recurrent neural network-basedclassifier and the trained non-recurrent neural network-basedclassifier.
 4. The system of claim 1, further configured to: for a giveninput, during inference, based on output probabilities of the traineddecision neural network-based classifier, perform the machineclassification task on the given input using either the trainedrecurrent neural network-based classifier or the trained non-recurrentneural network-based classifier.
 5. The system of claim 3, furtherconfigured to select the trained recurrent neural network-basedclassifier when the output probability of the first model class label ishigher than that of the second model class label.
 6. The system of claim3, further configured to select the trained non-recurrent neuralnetwork-based classifier when the output probability of the second modelclass label is higher than that of the first model class label.
 7. Thesystem of claim 1, wherein the trained recurrent neural network-basedclassifier is at least three percent more accurate and four timescomputationally more expensive than the trained non-recurrent neuralnetwork-based classifier.
 8. The system of claim 7, wherein the trainedrecurrent neural network-based classifier is at least one recurrentneural network (abbreviated RNN).
 9. The system of claim 7, wherein thetrained non-recurrent neural network-based classifier is at least onebag of words (abbreviated BoW) network.
 10. The system of claim 7,wherein the trained non-recurrent neural network-based classifier is atleast one continuous bag of words (abbreviated CBoW) network.
 11. Thesystem of claim 7, wherein the trained non-recurrent neuralnetwork-based classifier is at least one skip-gram network.
 12. Thesystem of claim 7, wherein the trained non-recurrent neuralnetwork-based classifier is at least one convolutional neural network(abbreviated CNN).
 13. The system of claim 8, wherein the RNN is a longshort-term memory (abbreviated LSTM) network.
 14. The system of claim 8,wherein the RNN is a gated recurrent unit (abbreviated GRU) network. 15.The system of claim 8, wherein the RNN is a quasi-recurrent neuralnetwork (abbreviated QRNN).
 16. The system of claim 3, wherein thetraining set and the validation set are part of a single data set thatis subjected to held-out splitting to create the training set and thevalidation set.
 17. The system of claim 3, wherein the trained recurrentneural network-based classifier and the trained non-recurrent neuralnetwork-based classifier are trained separately on the training set. 18.The system of claim 1, wherein the machine classification task issentiment classification and the inputs are sentences.
 19. The system ofclaim 18, wherein the task class labels are at least one of positivesentiment, negative sentiment, very positive sentiment, very negativesentiment, somewhat positive sentiment, somewhat negative sentiment, orneutral sentiment.
 20. The system of claim 3, wherein the confusionmatrix identifies at least one of: validation inputs accuratelyclassified by both the trained recurrent neural network-based classifierand the trained non-recurrent neural network-based classifier;validation inputs inaccurately classified by both the trained recurrentneural network-based classifier and the trained non-recurrent neuralnetwork-based classifier; validation inputs accurately classified by thetrained non-recurrent neural network-based classifier but inaccuratelyclassified by the trained recurrent neural network-based classifier; andvalidation inputs accurately classified by the trained recurrent neuralnetwork-based classifier but inaccurately classified by the trainednon-recurrent neural network-based classifier.
 21. The system of claim3, wherein the first subset includes validation inputs accuratelyclassified by the trained recurrent neural network-based classifier butinaccurately classified by the trained non-recurrent neuralnetwork-based classifier.
 22. The system of claim 3, wherein the secondsubset includes at least one of: validation inputs accurately classifiedby both the trained recurrent neural network-based classifier and thetrained non-recurrent neural network-based classifier; validation inputsinaccurately classified by both the trained recurrent neuralnetwork-based classifier and the trained non-recurrent neuralnetwork-based classifier; and validation inputs accurately classified bythe trained non-recurrent neural network-based classifier butinaccurately classified by the trained recurrent neural network-basedclassifier.
 23. The system of claim 1, wherein the trained recurrentneural network-based classifier and the trained non-recurrent neuralnetwork-based classifier selected by the trained decision neuralnetwork-based classifier during inference are trained on a combinationof the training set and the validation set.
 24. The system of claim 1,wherein the decision neural network-based classifier comprises thetrained non-recurrent neural network-based classifier trained on thetraining set, with an original classification layer ablated, one or morenew fully-connected layers, and a new classification layer that producesoutput probabilities for the first and second model class labels. 25.The system of claim 24, further configured to, during training,backpropagate gradients only for the fully-connected layers and the newclassification layer and keeping weights of the trained non-recurrentneural network-based classifier fixed.
 26. The system of claim 24,wherein the trained non-recurrent neural network-based classifier is atleast one bag of words (abbreviated BoW) network.
 27. The system ofclaim 1, wherein the machine classification task is at least one of:part-of-speech (abbreviated POS) tagging, chunking, dependency parsing,semantic relatedness, and textual entailment.
 28. The system of claim 1,wherein the machine classification task is at least one of: speechrecognition, machine translation, text summarization, questionanswering, image captioning, and text-to-speech (abbreviated TTS)synthesis.
 29. A neural network-based decision system with processorsoperating in parallel to efficiently perform a sentiment classificationtask on a sentence using either a trained recurrent long short-termmemory (abbreviated LSTM) network or a trained bag of words (abbreviatedBoW) network that generates a mean vector representation of a sentenceby averaging token vector embeddings of the sentence, the systemimplementing actions comprising: generating a confusion matrix based onthe trained LSTM and BoW networks' evaluation of validation sentencesannotated with positive and negative sentiment labels; using theconfusion matrix to identify a subset of validation sentences accuratelyclassified only by the trained LSTM network and annotating the subset ofvalidation sentences with a first model label identifying the trainedLSTM network, annotating remaining of the validation sentences with asecond model label identifying the trained BoW network, and storing themodel-annotated validation sentences in a decision set; and constructinga decision system using the trained BoW network and training thedecision system using the decision set to produce an output thatspecifies whether to use the trained LSTM network or the trained BoWnetwork for classifying the sentence's sentiment.
 30. The system ofclaim 29, further including, during inference, using an output of thetrained decision system to select either the trained LSTM network or thetrained BoW network to classify a sentence's sentiment.
 31. The systemof claim 30, wherein the decision system selects the trained LSTMnetwork when the sentence is a linguistically complex sentence.
 32. Thesystem of claim 30, wherein the decision system selects the trained BoWnetwork when the sentence is a linguistically simple sentence.
 33. Anon-transitory, computer-readable medium having computer executableinstructions that implement the system of claim
 1. 34. A non-transitory,computer-readable medium having computer executable instructions thatimplement the system of claim 29.